Image encoding/decoding method and device, and method for transmitting bitstream

ABSTRACT

An image encoding/decoding method and apparatus are provided. An image decoding method according to the present disclosure may include determining whether a prediction mode of a current block is an intra prediction mode based on information on the prediction mode of the current block, determining whether intra sub-partitions (ISP) is available for the current block, when the prediction mode of the current block is an intra prediction mode, decoding an ISP application indicator indicating whether to apply ISP to the current block, when ISP is available for the current block, and generating a prediction block for the current block by applying ISP to the current block, upon determining that ISP applies to the current block based on the ISP application indicator.

This application is a Bypass National Stage filing under 35 U.S.C. 371of International Application No. PCT/KR2020/003359, filed on Mar. 11,2020, which claims the benefit of U.S. Patent Application No. 62/817,580filed on Mar. 13, 2019, the contents of which are all herebyincorporated by reference herein in their entirety.

TECHNICAL FIELD

The present disclosure relates to an image encoding/decoding method andapparatus, and a method of transmitting a bitstream, and, moreparticularly, to a method and apparatus for encoding/decoding an imageby applying intra subblock-partitions (ISP) to images having variouschroma formats, and a method of transmitting a bitstream generated bythe image encoding method/apparatus of the present disclosure.

BACKGROUND ART

Recently, demand for high-resolution and high-quality images such ashigh definition (HD) images and ultra high definition (UHD) images isincreasing in various fields. As resolution and quality of image dataare improved, the amount of transmitted information or bits relativelyincreases as compared to existing image data. An increase in the amountof transmitted information or bits causes an increase in transmissioncost and storage cost.

Accordingly, there is a need for high-efficient image compressiontechnology for effectively transmitting, storing and reproducinginformation on high-resolution and high-quality images.

DISCLOSURE Technical Problem

An object of the present disclosure is to provide an imageencoding/decoding method and apparatus with improved encoding/decodingefficiency.

An object of the present disclosure is to provide a method and apparatusfor encoding/decoding an image by applying ISP to images having variouschroma formats.

Another object of the present disclosure is to provide a method oftransmitting a bitstream generated by an image encoding method orapparatus according to the present disclosure.

Another object of the present disclosure is to provide a recordingmedium storing a bitstream generated by an image encoding method orapparatus according to the present disclosure.

Another object of the present disclosure is to provide a recordingmedium storing a bitstream received, decoded and used to reconstruct animage by an image decoding apparatus according to the presentdisclosure.

The technical problems solved by the present disclosure are not limitedto the above technical problems and other technical problems which arenot described herein will become apparent to those skilled in the artfrom the following description.

Technical Solution

An image decoding method performed by an image decoding apparatusaccording to an aspect of the present disclosure may include determiningwhether a prediction mode of a current block is an intra prediction modebased on information on the prediction mode of the current block,determining whether intra sub-partitions (ISP) is available for thecurrent block, when the prediction mode of the current block is an intraprediction mode, decoding an ISP application indicator indicatingwhether to apply ISP to the current block, when ISP is available for thecurrent block, and generating a prediction block for the current blockby applying ISP to the current block, upon determining that ISP appliesto the current block based on the ISP application indicator.

In the image decoding method of the present disclosure, the determiningwhether ISP is available for the current block may be performed based oncomparison between a size of a luma component block of the current blockand a predetermined threshold.

In the image decoding method of the present disclosure, thepredetermined threshold may be a maximum transform size.

In the image decoding method of the present disclosure, when both awidth and height of the luma component block of the current block areequal to or less than a predetermined threshold, it may be determinedthat ISP is available for the current block.

In the image decoding method of the present disclosure, the generatingthe prediction block for the current block by applying ISP to thecurrent block may include generating a prediction block for a lumacomponent block by applying ISP to the luma component block of thecurrent block, determining whether to apply ISP to a chroma componentblock of the current block, and generating the prediction block for thechroma component block by applying ISP to the chroma component block,upon determining that ISP applies to the chroma component block of thecurrent block.

In the image decoding method of the present disclosure, the determiningwhether to apply ISP to the chroma component block of the current blockmay be performed based on at least one of a size of the chroma componentblock or a chroma format of the current block.

In the image decoding method of the present disclosure, when the chromaformat of the current block is 4:2:2 or 4:4:4 and at least one of awidth or height of the chroma component block is greater than a maximumtransform size, it may be determined that ISP applies to the chromacomponent block of the current block.

In the image decoding method of the present disclosure, when ISP appliesto the chroma component block, a split direction and the number ofsub-partitions for the chroma component block may be determined based ona split direction and the number of sub-partitions for the lumacomponent block.

In the image decoding method of the present disclosure, the splitdirection and the number of sub-partitions for the chroma componentblock may be equal to the split direction and the number ofsub-partitions for the luma component block, respectively.

In the image decoding method of the present disclosure, the generatingthe prediction block for the current block by applying ISP to thecurrent block may include generating a prediction block for a lumacomponent block by applying ISP to the luma component block of thecurrent block and generating a prediction block for a chroma componentblock by applying ISP to the chroma component block of the currentblock.

In the image decoding method of the present disclosure, the splitdirection and the number of sub-partitions for the chroma componentblock may be determined based on the split direction and the number ofsub-partitions for the luma component block.

In the image decoding method of the present disclosure, the splitdirection and the number of sub-partitions for the chroma componentblock may be equal to the split direction and (the number ofsub-partitions/2) of the luma component block, respectively.

An image decoding apparatus according to another aspect of the presentdisclosure may include a memory and at least one processor. At least oneprocessor may determine whether a prediction mode of a current block isan intra prediction mode based on information on the prediction mode ofthe current block, determine whether intra sub-partitions (ISP) isavailable for the current block, when the prediction mode of the currentblock is an intra prediction mode, decode an ISP application indicatorindicating whether to apply ISP to the current block, when ISP isavailable for the current block, and generate a prediction block for thecurrent block by applying ISP to the current block, upon determiningthat ISP applies to the current block based on the ISP applicationindicator.

An image encoding method according to another aspect of the presentdisclosure may include determining whether a prediction mode of acurrent block is an intra prediction mode, determining whether intrasub-partitions (ISP) is available for the current block, when theprediction mode of the current block is an intra prediction mode,determining whether to apply ISP to the current block, when ISP isavailable for the current block, generating a prediction block for thecurrent block by applying ISP to the current block, when ISP applies tothe current block, and encoding information on the prediction mode ofthe current block and information on whether to apply ISP to the currentblock.

A transmission method according to another aspect of the presentdisclosure may transmit the bitstream generated by the image encodingapparatus or the image encoding method of the present disclosure.

A computer-readable recording medium according to another aspect of thepresent disclosure may store the bitstream generated by the imageencoding apparatus or the image encoding method of the presentdisclosure.

A computer-readable recording medium according to another aspect of thepresent disclosure may store a bitstream received and decoded by animage decoding apparatus and used to reconstruct an image. The bitstreammay include information on a prediction mode of a current block and anISP application indicator indicating whether to apply ISP to the currentblock, the information on the prediction mode of the current block maybe used to determine whether the prediction mode of the current block isan intra prediction mode, the ISP application indicator may be used todetermine whether to apply ISP to the current block when the predictionmode of the current block is an intra prediction mode and it isdetermined that ISP is available for the current block, and theprediction block of the current block may be generated by performing ISPwith respect to the current block upon determining that ISP applies tothe current block.

The features briefly summarized above with respect to the presentdisclosure are merely exemplary aspects of the detailed descriptionbelow of the present disclosure, and do not limit the scope of thepresent disclosure.

Advantageous Effects

According to the present disclosure, it is possible to provide an imageencoding/decoding method and apparatus with improved encoding/decodingefficiency.

Also, according to the present disclosure, it is possible to provide amethod and apparatus for encoding/decoding an image by applying ISP toimages having various chroma formats.

Also, according to the present disclosure, it is possible to provide amethod of transmitting a bitstream generated by an image encoding methodor apparatus according to the present disclosure.

Also, according to the present disclosure, it is possible to provide arecording medium storing a bitstream generated by an image encodingmethod or apparatus according to the present disclosure.

Also, according to the present disclosure, it is possible to provide arecording medium storing a bitstream received, decoded and used toreconstruct an image by an image decoding apparatus according to thepresent disclosure.

It will be appreciated by persons skilled in the art that that theeffects that can be achieved through the present disclosure are notlimited to what has been particularly described hereinabove and otheradvantages of the present disclosure will be more clearly understoodfrom the detailed description.

DESCRIPTION OF DRAWINGS

FIG. 1 is a view schematically showing a video coding system, to whichan embodiment of the present disclosure is applicable.

FIG. 2 is a view schematically showing an image encoding apparatus, towhich an embodiment of the present disclosure is applicable.

FIG. 3 is a view schematically showing an image decoding apparatus, towhich an embodiment of the present disclosure is applicable.

FIG. 4 is a flowchart illustrating an intra prediction based video/imageencoding method.

FIG. 5 is a view illustrating the configuration of an intra predictor185 according to the present disclosure.

FIG. 6 is a flowchart illustrating an intra prediction based video/imagedecoding method.

FIG. 7 is a view illustrating the configuration of an intra predictor265 according to the present disclosure.

FIGS. 8 a and 8 b are views illustrating an intra prediction directionaccording to an embodiment of the present disclosure.

FIG. 9 is a view illustrating a conventional method of performingencoding/decoding of a current block according to an ISP mode.

FIGS. 10 a and 10 b are views illustrating a split example of ISP forcurrent blocks having various sizes.

FIGS. 11 a, 11 b, and 11 c are views illustrating a relationship betweena luma component block (luma component array) and a chroma componentblock (chroma component array) according to a chroma format.

FIG. 12 is a view illustrating the size of a chroma component blockaccording to a chroma format when a luma component block is a 64×128block.

FIG. 13 is a flowchart illustrating an ISP application method accordingto an embodiment of the present disclosure.

FIG. 14 is a view illustrating an example of splitting a chromacomponent block according to the ISP application method of FIG. 13 .

FIG. 15 is a flowchart illustrating an ISP application method accordingto another embodiment of the present disclosure.

FIG. 16 is a view illustrating an example of splitting a chromacomponent block according to the ISP application method of FIG. 15 .

FIG. 17 is a view illustrating an example of the structure of abitstream in which a modified ISP availability condition is reflected.

FIG. 18 is a view illustrating another example of the structure of abitstream in which a modified ISP availability condition is reflected.

FIG. 19 is a view showing a content streaming system, to which anembodiment of the present disclosure is applicable.

MODE FOR INVENTION

Hereinafter, the embodiments of the present disclosure will be describedin detail with reference to the accompanying drawings so as to be easilyimplemented by those skilled in the art. However, the present disclosuremay be implemented in various different forms, and is not limited to theembodiments described herein.

In describing the present disclosure, if it is determined that thedetailed description of a related known function or construction rendersthe scope of the present disclosure unnecessarily ambiguous, thedetailed description thereof will be omitted. In the drawings, parts notrelated to the description of the present disclosure are omitted, andsimilar reference numerals are attached to similar parts.

In the present disclosure, when a component is “connected”, “coupled” or“linked” to another component, it may include not only a directconnection relationship but also an indirect connection relationship inwhich an intervening component is present. In addition, when a component“includes” or “has” other components, it means that other components maybe further included, rather than excluding other components unlessotherwise stated.

In the present disclosure, the terms first, second, etc. may be usedonly for the purpose of distinguishing one component from othercomponents, and do not limit the order or importance of the componentsunless otherwise stated. Accordingly, within the scope of the presentdisclosure, a first component in one embodiment may be referred to as asecond component in another embodiment, and similarly, a secondcomponent in one embodiment may be referred to as a first component inanother embodiment.

In the present disclosure, components that are distinguished from eachother are intended to clearly describe each feature, and do not meanthat the components are necessarily separated. That is, a plurality ofcomponents may be integrated and implemented in one hardware or softwareunit, or one component may be distributed and implemented in a pluralityof hardware or software units. Therefore, even if not stated otherwise,such embodiments in which the components are integrated or the componentis distributed are also included in the scope of the present disclosure.

In the present disclosure, the components described in variousembodiments do not necessarily mean essential components, and somecomponents may be optional components. Accordingly, an embodimentconsisting of a subset of components described in an embodiment is alsoincluded in the scope of the present disclosure. In addition,embodiments including other components in addition to componentsdescribed in the various embodiments are included in the scope of thepresent disclosure.

The present disclosure relates to encoding and decoding of an image, andterms used in the present disclosure may have a general meaning commonlyused in the technical field, to which the present disclosure belongs,unless newly defined in the present disclosure.

In the present disclosure, a “picture” generally refers to a unitrepresenting one image in a specific time period, and a slice/tile is acoding unit constituting a part of a picture, and one picture may becomposed of one or more slices/tiles. In addition, a slice/tile mayinclude one or more coding tree units (CTUs).

In the present disclosure, a “pixel” or a “pel” may mean a smallest unitconstituting one picture (or image). In addition, “sample” may be usedas a term corresponding to a pixel. A sample may generally represent apixel or a value of a pixel, and may represent only a pixel/pixel valueof a luma component or only a pixel/pixel value of a chroma component.

In the present disclosure, a “unit” may represent a basic unit of imageprocessing. The unit may include at least one of a specific region ofthe picture and information related to the region. The unit may be usedinterchangeably with terms such as “sample array”, “block” or “area” insome cases. In a general case, an M×N block may include samples (orsample arrays) or a set (or array) of transform coefficients of Mcolumns and N rows.

In the present disclosure, “current block” may mean one of “currentcoding block”, “current coding unit”, “coding target block”, “decodingtarget block” or “processing target block”. When prediction isperformed, “current block” may mean “current prediction block” or“prediction target block”. When transform (inversetransform)/quantization (dequantization) is performed, “current block”may mean “current transform block” or “transform target block”. Whenfiltering is performed, “current block” may mean “filtering targetblock”.

In the present disclosure, the term “/” and “,” should be interpreted toindicate “and/or.” For instance, the expression “A/B” and “A, B” maymean “A and/or B.” Further, “A/B/C” and “A/B/C” may mean “at least oneof A, B, and/or C.”

In the present disclosure, the term “or” should be interpreted toindicate “and/or.” For instance, the expression “A or B” may comprise 1)only “A”, 2) only “B”, and/or 3) both “A and B”. In other words, in thepresent disclosure, the term “or” should be interpreted to indicate“additionally or alternatively.”

Overview of Video Coding System

FIG. 1 is a view showing a video coding system according to the presentdisclosure.

The video coding system according to an embodiment may include aencoding apparatus 10 and a decoding apparatus 20. The encodingapparatus 10 may deliver encoded video and/or image information or datato the decoding apparatus 20 in the form of a file or streaming via adigital storage medium or network.

The encoding apparatus 10 according to an embodiment may include a videosource generator 11, an encoding unit 12 and a transmitter 13. Thedecoding apparatus 20 according to an embodiment may include a receiver21, a decoding unit 22 and a renderer 23. The encoding unit 12 may becalled a video/image encoding unit, and the decoding unit 22 may becalled a video/image decoding unit. The transmitter 13 may be includedin the encoding unit 12. The receiver 21 may be included in the decodingunit 22. The renderer 23 may include a display and the display may beconfigured as a separate device or an external component.

The video source generator 11 may acquire a video/image through aprocess of capturing, synthesizing or generating the video/image. Thevideo source generator 11 may include a video/image capture deviceand/or a video/image generating device. The video/image capture devicemay include, for example, one or more cameras, video/image archivesincluding previously captured video/images, and the like. Thevideo/image generating device may include, for example, computers,tablets and smartphones, and may (electronically) generate video/images.For example, a virtual video/image may be generated through a computeror the like. In this case, the video/image capturing process may bereplaced by a process of generating related data.

The encoding unit 12 may encode an input video/image. The encoding unit12 may perform a series of procedures such as prediction, transform, andquantization for compression and coding efficiency. The encoding unit 12may output encoded data (encoded video/image information) in the form ofa bitstream.

The transmitter 13 may transmit the encoded video/image information ordata output in the form of a bitstream to the receiver 21 of thedecoding apparatus 20 through a digital storage medium or a network inthe form of a file or streaming The digital storage medium may includevarious storage mediums such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, andthe like. The transmitter 13 may include an element for generating amedia file through a predetermined file format and may include anelement for transmission through a broadcast/communication network. Thereceiver 21 may extract/receive the bitstream from the storage medium ornetwork and transmit the bitstream to the decoding unit 22.

The decoding unit 22 may decode the video/image by performing a seriesof procedures such as dequantization, inverse transform, and predictioncorresponding to the operation of the encoding unit 12.

The renderer 23 may render the decoded video/image. The renderedvideo/image may be displayed through the display.

Overview of Image Encoding Apparatus

FIG. 2 is a view schematically showing an image encoding apparatus, towhich an embodiment of the present disclosure is applicable.

As shown in FIG. 2 , the image encoding apparatus 100 may include animage partitioner 110, a subtractor 115, a transformer 120, a quantizer130, a dequantizer 140, an inverse transformer 150, an adder 155, afilter 160, a memory 170, an inter predictor 180, an intra predictor 185and an entropy encoder 190. The inter predictor 180 and the intrapredictor 185 may be collectively referred to as a “predictor”. Thetransformer 120, the quantizer 130, the dequantizer 140 and the inversetransformer 150 may be included in a residual processor. The residualprocessor may further include the subtractor 115.

All or at least some of the plurality of components configuring theimage encoding apparatus 100 may be configured by one hardware component(e.g., an encoder or a processor) in some embodiments. In addition, thememory 170 may include a decoded picture buffer (DPB) and may beconfigured by a digital storage medium.

The image partitioner 110 may partition an input image (or a picture ora frame) input to the image encoding apparatus 100 into one or moreprocessing units. For example, the processing unit may be called acoding unit (CU). The coding unit may be acquired by recursivelypartitioning a coding tree unit (CTU) or a largest coding unit (LCU)according to a quad-tree binary-tree ternary-tree (QT/BT/TT) structure.For example, one coding unit may be partitioned into a plurality ofcoding units of a deeper depth based on a quad tree structure, a binarytree structure, and/or a ternary structure. For partitioning of thecoding unit, a quad tree structure may be applied first and the binarytree structure and/or ternary structure may be applied later. The codingprocedure according to the present disclosure may be performed based onthe final coding unit that is no longer partitioned. The largest codingunit may be used as the final coding unit or the coding unit of deeperdepth acquired by partitioning the largest coding unit may be used asthe final coding unit. Here, the coding procedure may include aprocedure of prediction, transform, and reconstruction, which will bedescribed later. As another example, the processing unit of the codingprocedure may be a prediction unit (PU) or a transform unit (TU). Theprediction unit and the transform unit may be split or partitioned fromthe final coding unit. The prediction unit may be a unit of sampleprediction, and the transform unit may be a unit for deriving atransform coefficient and/or a unit for deriving a residual signal fromthe transform coefficient.

The predictor (the inter predictor 180 or the intra predictor 185) mayperform prediction on a block to be processed (current block) andgenerate a predicted block including prediction samples for the currentblock. The predictor may determine whether intra prediction or interprediction is applied on a current block or CU basis. The predictor maygenerate various information related to prediction of the current blockand transmit the generated information to the entropy encoder 190. Theinformation on the prediction may be encoded in the entropy encoder 190and output in the form of a bitstream.

The intra predictor 185 may predict the current block by referring tothe samples in the current picture. The referred samples may be locatedin the neighborhood of the current block or may be located apartaccording to the intra prediction mode and/or the intra predictiontechnique. The intra prediction modes may include a plurality ofnon-directional modes and a plurality of directional modes. Thenon-directional mode may include, for example, a DC mode and a planarmode. The directional mode may include, for example, 33 directionalprediction modes or 65 directional prediction modes according to thedegree of detail of the prediction direction. However, this is merely anexample, more or less directional prediction modes may be used dependingon a setting. The intra predictor 185 may determine the prediction modeapplied to the current block by using a prediction mode applied to aneighboring block.

The inter predictor 180 may derive a predicted block for the currentblock based on a reference block (reference sample array) specified by amotion vector on a reference picture. In this case, in order to reducethe amount of motion information transmitted in the inter predictionmode, the motion information may be predicted in units of blocks,subblocks, or samples based on correlation of motion information betweenthe neighboring block and the current block. The motion information mayinclude a motion vector and a reference picture index. The motioninformation may further include inter prediction direction (L0prediction, L1 prediction, Bi prediction, etc.) information. In the caseof inter prediction, the neighboring block may include a spatialneighboring block present in the current picture and a temporalneighboring block present in the reference picture. The referencepicture including the reference block and the reference pictureincluding the temporal neighboring block may be the same or different.The temporal neighboring block may be called a collocated referenceblock, a co-located CU (colCU), and the like. The reference pictureincluding the temporal neighboring block may be called a collocatedpicture (colPic). For example, the inter predictor 180 may configure amotion information candidate list based on neighboring blocks andgenerate information indicating which candidate is used to derive amotion vector and/or a reference picture index of the current block.Inter prediction may be performed based on various prediction modes. Forexample, in the case of a skip mode and a merge mode, the interpredictor 180 may use motion information of the neighboring block asmotion information of the current block. In the case of the skip mode,unlike the merge mode, the residual signal may not be transmitted. Inthe case of the motion vector prediction (MVP) mode, the motion vectorof the neighboring block may be used as a motion vector predictor, andthe motion vector of the current block may be signaled by encoding amotion vector difference and an indicator for a motion vector predictor.The motion vector difference may mean a difference between the motionvector of the current block and the motion vector predictor.

The predictor may generate a prediction signal based on variousprediction methods and prediction techniques described below. Forexample, the predictor may not only apply intra prediction or interprediction but also simultaneously apply both intra prediction and interprediction, in order to predict the current block. A prediction methodof simultaneously applying both intra prediction and inter predictionfor prediction of the current block may be called combined inter andintra prediction (CIIP). In addition, the predictor may perform intrablock copy (IBC) for prediction of the current block. Intra block copymay be used for content image/video coding of a game or the like, forexample, screen content coding (SCC). IBC is a method of predicting acurrent picture using a previously reconstructed reference block in thecurrent picture at a location apart from the current block by apredetermined distance. When IBC is applied, the location of thereference block in the current picture may be encoded as a vector (blockvector) corresponding to the predetermined distance.

The prediction signal generated by the predictor may be used to generatea reconstructed signal or to generate a residual signal. The subtractor115 may generate a residual signal (residual block or residual samplearray) by subtracting the prediction signal (predicted block orprediction sample array) output from the predictor from the input imagesignal (original block or original sample array). The generated residualsignal may be transmitted to the transformer 120.

The transformer 120 may generate transform coefficients by applying atransform technique to the residual signal. For example, the transformtechnique may include at least one of a discrete cosine transform (DCT),a discrete sine transform (DST), a karhunen-loève transform (KLT), agraph-based transform (GBT), or a conditionally non-linear transform(CNT). Here, the GBT means transform obtained from a graph whenrelationship information between pixels is represented by the graph. TheCNT refers to transform acquired based on a prediction signal generatedusing all previously reconstructed pixels. In addition, the transformprocess may be applied to square pixel blocks having the same size ormay be applied to blocks having a variable size rather than square.

The quantizer 130 may quantize the transform coefficients and transmitthem to the entropy encoder 190. The entropy encoder 190 may encode thequantized signal (information on the quantized transform coefficients)and output a bitstream. The information on the quantized transformcoefficients may be referred to as residual information. The quantizer130 may rearrange quantized transform coefficients in a block form intoa one-dimensional vector form based on a coefficient scanning order andgenerate information on the quantized transform coefficients based onthe quantized transform coefficients in the one-dimensional vector form.

The entropy encoder 190 may perform various encoding methods such as,for example, exponential Golomb, context-adaptive variable length coding(CAVLC), context-adaptive binary arithmetic coding (CABAC), and thelike. The entropy encoder 190 may encode information necessary forvideo/image reconstruction other than quantized transform coefficients(e.g., values of syntax elements, etc.) together or separately. Encodedinformation (e.g., encoded video/image information) may be transmittedor stored in units of network abstraction layers (NALs) in the form of abitstream. The video/image information may further include informationon various parameter sets such as an adaptation parameter set (APS), apicture parameter set (PPS), a sequence parameter set (SPS), or a videoparameter set (VPS). In addition, the video/image information mayfurther include general constraint information. The signaledinformation, transmitted information and/or syntax elements described inthe present disclosure may be encoded through the above-describedencoding procedure and included in the bitstream.

The bitstream may be transmitted over a network or may be stored in adigital storage medium. The network may include a broadcasting networkand/or a communication network, and the digital storage medium mayinclude various storage media such as USB, SD, CD, DVD, Blu-ray, HDD,SSD, and the like. A transmitter (not shown) transmitting a signaloutput from the entropy encoder 190 and/or a storage unit (not shown)storing the signal may be included as internal/external element of theimage encoding apparatus 100. Alternatively, the transmitter may beprovided as the component of the entropy encoder 190.

The quantized transform coefficients output from the quantizer 130 maybe used to generate a residual signal. For example, the residual signal(residual block or residual samples) may be reconstructed by applyingdequantization and inverse transform to the quantized transformcoefficients through the dequantizer 140 and the inverse transformer150.

The adder 155 adds the reconstructed residual signal to the predictionsignal output from the inter predictor 180 or the intra predictor 185 togenerate a reconstructed signal (reconstructed picture, reconstructedblock, reconstructed sample array). If there is no residual for theblock to be processed, such as a case where the skip mode is applied,the predicted block may be used as the reconstructed block. The adder155 may be called a reconstructor or a reconstructed block generator.The generated reconstructed signal may be used for intra prediction of anext block to be processed in the current picture and may be used forinter prediction of a next picture through filtering as described below.

Meanwhile, as described below, luma mapping with chroma scaling (MCS) isapplicable in a picture encoding process.

The filter 160 may improve subjective/objective image quality byapplying filtering to the reconstructed signal. For example, the filter160 may generate a modified reconstructed picture by applying variousfiltering methods to the reconstructed picture and store the modifiedreconstructed picture in the memory 170, specifically, a DPB of thememory 170. The various filtering methods may include, for example,deblocking filtering, a sample adaptive offset, an adaptive loop filter,a bilateral filter, and the like. The filter 160 may generate variousinformation related to filtering and transmit the generated informationto the entropy encoder 190 as described later in the description of eachfiltering method. The information related to filtering may be encoded bythe entropy encoder 190 and output in the form of a bitstream.

The modified reconstructed picture transmitted to the memory 170 may beused as the reference picture in the inter predictor 180. When interprediction is applied through the image encoding apparatus 100,prediction mismatch between the image encoding apparatus 100 and theimage decoding apparatus may be avoided and encoding efficiency may beimproved.

The DPB of the memory 170 may store the modified reconstructed picturefor use as a reference picture in the inter predictor 180. The memory170 may store the motion information of the block from which the motioninformation in the current picture is derived (or encoded) and/or themotion information of the blocks in the picture that have already beenreconstructed. The stored motion information may be transmitted to theinter predictor 180 and used as the motion information of the spatialneighboring block or the motion information of the temporal neighboringblock. The memory 170 may store reconstructed samples of reconstructedblocks in the current picture and may transfer the reconstructed samplesto the intra predictor 185.

Overview of Image Decoding Apparatus

FIG. 3 is a view schematically showing an image decoding apparatus, towhich an embodiment of the present disclosure is applicable.

As shown in FIG. 3 , the image decoding apparatus 200 may include anentropy decoder 210, a dequantizer 220, an inverse transformer 230, anadder 235, a filter 240, a memory 250, an inter predictor 260 and anintra predictor 265. The inter predictor 260 and the intra predictor 265may be collectively referred to as a “predictor”. The dequantizer 220and the inverse transformer 230 may be included in a residual processor.

All or at least some of a plurality of components configuring the imagedecoding apparatus 200 may be configured by a hardware component (e.g.,a decoder or a processor) according to an embodiment. In addition, thememory 250 may include a decoded picture buffer (DPB) or may beconfigured by a digital storage medium.

The image decoding apparatus 200, which has received a bitstreamincluding video/image information, may reconstruct an image byperforming a process corresponding to a process performed by the imageencoding apparatus 100 of FIG. 2 . For example, the image decodingapparatus 200 may perform decoding using a processing unit applied inthe image encoding apparatus. Thus, the processing unit of decoding maybe a coding unit, for example. The coding unit may be acquired bypartitioning a coding tree unit or a largest coding unit. Thereconstructed image signal decoded and output through the image decodingapparatus 200 may be reproduced through a reproducing apparatus (notshown).

The image decoding apparatus 200 may receive a signal output from theimage encoding apparatus of FIG. 2 in the form of a bitstream. Thereceived signal may be decoded through the entropy decoder 210. Forexample, the entropy decoder 210 may parse the bitstream to deriveinformation (e.g., video/image information) necessary for imagereconstruction (or picture reconstruction). The video/image informationmay further include information on various parameter sets such as anadaptation parameter set (APS), a picture parameter set (PPS), asequence parameter set (SPS), or a video parameter set (VPS). Inaddition, the video/image information may further include generalconstraint information. The image decoding apparatus may further decodepicture based on the information on the parameter set and/or the generalconstraint information. Signaled/received information and/or syntaxelements described in the present disclosure may be decoded through thedecoding procedure and obtained from the bitstream. For example, theentropy decoder 210 decodes the information in the bitstream based on acoding method such as exponential Golomb coding, CAVLC, or CABAC, andoutput values of syntax elements required for image reconstruction andquantized values of transform coefficients for residual. Morespecifically, the CABAC entropy decoding method may receive a bincorresponding to each syntax element in the bitstream, determine acontext model using a decoding target syntax element information,decoding information of a neighboring block and a decoding target blockor information of a symbol/bin decoded in a previous stage, and performarithmetic decoding on the bin by predicting a probability of occurrenceof a bin according to the determined context model, and generate asymbol corresponding to the value of each syntax element. In this case,the CABAC entropy decoding method may update the context model by usingthe information of the decoded symbol/bin for a context model of a nextsymbol/bin after determining the context model. The information relatedto the prediction among the information decoded by the entropy decoder210 may be provided to the predictor (the inter predictor 260 and theintra predictor 265), and the residual value on which the entropydecoding was performed in the entropy decoder 210, that is, thequantized transform coefficients and related parameter information, maybe input to the dequantizer 220. In addition, information on filteringamong information decoded by the entropy decoder 210 may be provided tothe filter 240. Meanwhile, a receiver (not shown) for receiving a signaloutput from the image encoding apparatus may be further configured as aninternal/external element of the image decoding apparatus 200, or thereceiver may be a component of the entropy decoder 210.

Meanwhile, the image decoding apparatus according to the presentdisclosure may be referred to as a video/image/picture decodingapparatus. The image decoding apparatus may be classified into aninformation decoder (video/image/picture information decoder) and asample decoder (video/image/picture sample decoder). The informationdecoder may include the entropy decoder 210. The sample decoder mayinclude at least one of the dequantizer 220, the inverse transformer230, the adder 235, the filter 240, the memory 250, the inter predictor160 or the intra predictor 265.

The dequantizer 220 may dequantize the quantized transform coefficientsand output the transform coefficients. The dequantizer 220 may rearrangethe quantized transform coefficients in the form of a two-dimensionalblock. In this case, the rearrangement may be performed based on thecoefficient scanning order performed in the image encoding apparatus.The dequantizer 220 may perform dequantization on the quantizedtransform coefficients by using a quantization parameter (e.g.,quantization step size information) and obtain transform coefficients.

The inverse transformer 230 may inversely transform the transformcoefficients to obtain a residual signal (residual block, residualsample array).

The predictor may perform prediction on the current block and generate apredicted block including prediction samples for the current block. Thepredictor may determine whether intra prediction or inter prediction isapplied to the current block based on the information on the predictionoutput from the entropy decoder 210 and may determine a specificintra/inter prediction mode (prediction technique).

It is the same as described in the predictor of the image encodingapparatus 100 that the predictor may generate the prediction signalbased on various prediction methods (techniques) which will be describedlater.

The intra predictor 265 may predict the current block by referring tothe samples in the current picture. The description of the intrapredictor 185 is equally applied to the intra predictor 265.

The inter predictor 260 may derive a predicted block for the currentblock based on a reference block (reference sample array) specified by amotion vector on a reference picture. In this case, in order to reducethe amount of motion information transmitted in the inter predictionmode, motion information may be predicted in units of blocks, subblocks,or samples based on correlation of motion information between theneighboring block and the current block. The motion information mayinclude a motion vector and a reference picture index. The motioninformation may further include inter prediction direction (L0prediction, L1 prediction, Bi prediction, etc.) information. In the caseof inter prediction, the neighboring block may include a spatialneighboring block present in the current picture and a temporalneighboring block present in the reference picture. For example, theinter predictor 260 may configure a motion information candidate listbased on neighboring blocks and derive a motion vector of the currentblock and/or a reference picture index based on the received candidateselection information. Inter prediction may be performed based onvarious prediction modes, and the information on the prediction mayinclude information indicating a mode of inter prediction for thecurrent block.

The adder 235 may generate a reconstructed signal (reconstructedpicture, reconstructed block, reconstructed sample array) by adding theobtained residual signal to the prediction signal (predicted block,predicted sample array) output from the predictor (including the interpredictor 260 and/or the intra predictor 265). The description of theadder 155 is equally applicable to the adder 235.

Meanwhile, as described below, luma mapping with chroma scaling (MCS) isapplicable in a picture decoding process.

The filter 240 may improve subjective/objective image quality byapplying filtering to the reconstructed signal. For example, the filter240 may generate a modified reconstructed picture by applying variousfiltering methods to the reconstructed picture and store the modifiedreconstructed picture in the memory 250, specifically, a DPB of thememory 250. The various filtering methods may include, for example,deblocking filtering, a sample adaptive offset, an adaptive loop filter,a bilateral filter, and the like.

The (modified) reconstructed picture stored in the DPB of the memory 250may be used as a reference picture in the inter predictor 260. Thememory 250 may store the motion information of the block from which themotion information in the current picture is derived (or decoded) and/orthe motion information of the blocks in the picture that have alreadybeen reconstructed. The stored motion information may be transmitted tothe inter predictor 260 so as to be utilized as the motion informationof the spatial neighboring block or the motion information of thetemporal neighboring block. The memory 250 may store reconstructedsamples of reconstructed blocks in the current picture and transfer thereconstructed samples to the intra predictor 265.

In the present disclosure, the embodiments described in the filter 160,the inter predictor 180, and the intra predictor 185 of the imageencoding apparatus 100 may be equally or correspondingly applied to thefilter 240, the inter predictor 260, and the intra predictor 265 of theimage decoding apparatus 200.

Overview of Intra Prediction

Hereinafter, intra prediction according to an embodiment will bedescribed.

Intra prediction may indicate prediction which generates predictionsamples for a current block based on reference samples in a picture towhich the current block belongs (hereinafter referred to as a currentpicture). When intra prediction applies to the current block,neighboring reference samples to be used for intra prediction of thecurrent block may be derived. The neighboring reference samples of thecurrent block may include a sample adjacent to a left boundary of thecurrent block having a size of nW×nH and a total of 2×nH samplesadjacent to the bottom-left, a sample adjacent to a top boundary of thecurrent block and a total of 2×nW samples adjacent to the top-right, andone sample adjacent to the top-left of the current block. Alternatively,the neighboring reference samples of the current block may include aplurality of columns of top neighboring samples and a plurality of rowsof left neighboring samples. In addition, the neighboring referencesamples of the current block may include a total of nH samples adjacentto a right boundary of the current block having a size of nW×nH, a totalof nW samples adjacent to a bottom boundary of the current block, andone sample adjacent to the bottom-right of the current block.

Some of the neighboring reference samples of the current block have notyet been decoded or may not be available. In this case, a decoder mayconstruct neighboring reference samples to be used for prediction, bysubstituting unavailable samples with available samples. Alternatively,neighboring reference samples to be used for prediction may beconstructed using interpolation of available samples.

When the neighboring reference samples are derived, (i) a predictionsample may be derived based on average or interpolation of neighboringreference samples of the current block and (ii) the prediction samplemay be derived based on a reference sample present in a specific(prediction) direction with respect to the prediction sample among theneighboring reference samples of the current block. The case of (i) maybe referred to as a non-directional mode or a non-angular mode and thecase of (ii) may be referred to as a directional mode or an angularmode.

In addition, the prediction sample may be generated throughinterpolation with a first neighboring sample located in a predictiondirection of the intra prediction mode of the current block and a secondneighboring sample located in the opposite direction based on aprediction target sample of the current block among the neighboringreference samples. The above-described case may be referred to as linearinterpolation intra prediction (LIP).

In addition, chroma prediction samples may be generated based on lumasamples using a linear model. This case may be called a linear model(LM) mode.

In addition, a temporary prediction sample of the current block may bederived based on filtered neighboring reference samples, and theprediction sample of the current block may be derived byweighted-summing the temporary prediction sample and at least onereference sample derived according to the intra prediction mode amongthe existing neighboring reference samples, that is, the unfilteredneighboring reference samples. This case may be referred to as positiondependent intra prediction (PDPC).

In addition, a reference sample line with highest prediction accuracymay be selected from multiple neighboring reference sample lines of thecurrent block to derive a prediction sample using a reference samplelocated in a prediction direction in the corresponding line, and, atthis time, information (e.g., intra_luma_ref_idx) on the used referencesample line may be encoded and signaled in a bitstream. This case may bereferred to as multi-reference line (MRL) intra prediction or MRL basedintra prediction.

In addition, the current block may be split into vertical or horizontalsub-partitions to perform intra prediction with respect to eachsub-partition based on the same intra prediction mode. At this time,neighboring reference samples of intra prediction may be derived inunits of sub-partitions. That is, a reconstructed sample of a previoussub-partition in encoding/decoding order may be used as a neighboringreference sample of a current sub-partition. In this case, the intraprediction mode for the current block equally applies to thesub-partitions and the neighboring reference samples are derived andused in units of sub-partitions, thereby increasing intra predictionperformance. Such a prediction method may be referred to as intrasub-partitions (ISP) or ISP based intra prediction.

The intra prediction technique may be referred to as various terms suchas intra prediction type or additional intra prediction mode to bedistinguished from a directional or non-directional intra predictionmode. For example, the intra prediction technique (intra prediction typeor the additional intra prediction mode) may include at least one ofLIP, LM, PDPC, MRL, ISP or MIP. Meanwhile, if necessary, post-filteringmay be performed with respect to the derived prediction sample.

Specifically, the intra prediction procedure may include an intraprediction mode/type determination step, a neighboring reference samplederivation step and an intra prediction mode/type based predictionsample derivation step. In addition, if necessary, post-filtering may beperformed with respect to the derived prediction sample.

FIG. 4 is a flowchart illustrating an intra prediction based video/imageencoding method.

The encoding method of FIG. 4 may be performed by the image encodingapparatus of FIG. 2 . Specifically, step S410 may be performed by theintra predictor 185, and step S420 may be performed by the residualprocessor. Specifically, step S420 may be performed by the subtractor115. Step S430 may be performed by the entropy encoder 190. Theprediction information of step S430 may be derived by the intrapredictor 185, and the residual information of step S430 may be derivedby the residual processor. The residual information is information onthe residual samples. The residual information may include informationon quantized transform coefficient for the residual samples. Asdescribed above, the residual samples may be derived as transformcoefficient through the transformer 120 of the image encoding apparatus,and the transform coefficient may be derived as the transformcoefficients quantized through the quantizer 130. The information on thequantized transform coefficients may be encoded by the entropy encoder190 through a residual coding procedure.

The image encoding apparatus may perform intra prediction with respectto a current block (S410). The image encoding apparatus may determine anintra prediction mode/type for the current block, derive neighboringreference samples of the current block, and generate prediction samplesin the current block based on the intra prediction mode/type and theneighboring reference samples. Here, the intra prediction mode/typedetermination, neighboring reference samples derivation and predictionsamples generation procedures may be simultaneously performed or any oneprocedure may be performed before the other procedures.

FIG. 5 is a view illustrating the configuration of an intra predictor185 according to the present disclosure.

As shown in FIG. 5 , the intra predictor 185 of the image encodingapparatus may include an intra prediction mode/type determination unit186, a reference sample derivation unit 187 and/or a prediction samplederivation unit 188. The intra prediction mode/type determination unit186 may determine an intra prediction mode/type for the current block.The reference sample derivation unit 187 may derive neighboringreference samples of the current block. The prediction sample derivationunit 188 may derive prediction samples of the current block. Meanwhile,although not shown, when the below-described prediction sample filteringprocedure is performed, the intra predictor 185 may further include aprediction sample filter (not shown).

The image encoding apparatus may determine a mode/type applying to thecurrent block among a plurality of intra prediction modes/types. Theimage encoding apparatus may compare rate distortion (RD) cost for theintra prediction modes/types and determine an optimal intra predictionmode/type for the current block.

Meanwhile, the image encoding apparatus may perform a prediction samplefiltering procedure. Prediction sample filtering may be referred to aspost-filtering. By the prediction sample filtering procedure, some orall of the prediction samples may be filtered. In some cases, theprediction sample filtering procedure may be omitted.

Referring to FIG. 4 again, the image encoding apparatus may generateresidual samples for the current block based on the prediction samplesor the filtered prediction samples (S420). The image encoding apparatusmay derive the residual samples by subtracting the prediction samplesfrom the original samples of the current block. That is, the imageencoding apparatus may derive the residual sample values by subtractingthe corresponding prediction sample value from the original samplevalue.

The image encoding apparatus may encode image information includinginformation on the intra prediction (prediction information) andresidual information of the residual samples. The prediction informationmay include the intra prediction mode information and/or the intraprediction technique information. The image encoding apparatus mayoutput the encoded image information in the form of a bitstream. Theoutput bitstream may be transmitted to the image decoding apparatusthrough a storage medium or a network.

The residual information may include residual coding syntax, which willbe described later. The image encoding apparatus may transform/quantizethe residual samples and derive quantized transform coefficients. Theresidual information may include information on the quantized transformcoefficients.

Meanwhile, as described above, the image encoding apparatus may generatea reconstructed picture (including reconstructed samples and areconstructed block). To this end, the image encoding apparatus mayperform dequantization/inverse transform with respect to the quantizedtransform coefficients and derive (modified) residual samples. Thereason for transforming/quantizing the residual samples and thenperforming dequantization/inverse transform is to derive the sameresidual samples as residual samples derived by the image decodingapparatus. The image encoding apparatus may generate a reconstructedbock including reconstructed samples for the current block based on theprediction samples and the (modified) residual samples. Based on thereconstructed block, the reconstructed picture for the current picturemay be generated. As described above, an in-loop filtering procedure isfurther applicable to the reconstructed picture.

FIG. 6 is a flowchart illustrating an intra prediction based video/imagedecoding method.

The image decoding apparatus may perform operation corresponding tooperation performed by the image encoding apparatus.

The decoding method of FIG. 6 may be performed by the image decodingapparatus of FIG. 3 . Steps S610 to S630 may be performed by the intrapredictor 265, and the prediction information of step S610 and theresidual information of step S640 may be obtained from a bitstream bythe entropy decoder 210. The residual processor of the image decodingapparatus may derive residual samples for the current block based on theresidual information (S640). Specifically, the dequantizer 220 of theresidual processor may perform dequantization based on the dequantizedtransform coefficients derived based on the residual information toderive transform coefficients, and the inverse transformer 230 of theresidual processor may perform inverse transform with respect to thetransform coefficients to derive the residual samples for the currentblock. Step S650 may be performed by the adder 235 or the reconstructor.

Specifically, the image decoding apparatus may derive an intraprediction mode/type for the current block based on the receivedprediction information (intra prediction mode/type information) (S610).The image decoding apparatus may derive neighboring reference samples ofthe current block (S620). The image decoding apparatus may generateprediction samples in the current block based on the intra predictionmode/type and the neighboring reference samples (S630). In this case,the image decoding apparatus may perform a prediction sample filteringprocedure. Prediction sample filtering may be referred to aspost-filtering. By the prediction sample filtering procedure, some orall of the prediction samples may be filtered. In some cases, theprediction sample filtering procedure may be omitted.

The image decoding apparatus may generate residual samples for thecurrent block based on the received residual information (S640). Theimage decoding apparatus may generate reconstructed samples for thecurrent block based on the prediction samples and the residual samplesand derive a reconstructed block including the reconstructed samples(S650). Based on the reconstructed block, the reconstructed picture forthe current picture may be generated. An in-loop filtering procedure isfurther applicable to the reconstructed picture, as described above.

FIG. 7 is a view illustrating the configuration of an intra predictor265 according to the present disclosure.

As shown in FIG. 7 , the intra predictor 265 of the image decodingapparatus may include an intra prediction mode/type determination unit266, a reference sample derivation unit 267 and a prediction samplederivation unit 268. The intra prediction mode/type determination unit266 may determine an intra prediction mode/type for the current blockbased on the intra prediction mode/type information generated andsignaled by the intra prediction mode/type determination unit 186 of theimage encoding apparatus, and the reference sample derivation unit 267may derive neighboring reference samples of the current block from areconstructed reference region in a current picture. The predictionsample derivation unit 268 may derive prediction samples of the currentblock. Meanwhile, although not shown, when the above-describedprediction sample filtering procedure is performed, the intra predictor265 may further include a prediction sample filter (not shown).

The intra prediction mode information may include, for example, flaginformation (e.g., intra_luma_mpm_flag) indicating whether to apply amost probable mode (MPM) or a remaining mode to the current block, and,when the MPM applies to the current block, the intra prediction modeinformation may further include index information (e.g.,intra_luma_mpm_idx) indicating one of the intra prediction modecandidates (MPM candidates). The intra prediction mode candidates (MPMcandidates) may be composed of an MPM candidate list or an MPM list. Inaddition, when the MPM does not apply to the current block, the intraprediction mode information may further include remaining modeinformation (e.g., intra_luma_mpm_remainder) indicating one of theremaining intra prediction modes excluding the intra prediction modecandidates (MPM candidates). The image decoding apparatus may determinethe intra prediction mode of the current block based on the intraprediction mode information. The MPM candidate modes may include theintra prediction modes of the neighboring blocks (e.g., the leftneighboring block and the upper neighboring block) of the current blockand additional candidate modes.

FIG. 8 a shows an intra prediction direction according to an embodimentof the present disclosure. In order to capture any edge directionpresented in natural video, as shown in FIG. 8 a, the intra predictionmode may include two non-directional intra prediction modes and 65directional intra prediction modes. The non-directional intra predictionmodes may include a planar intra prediction mode (planar mode) and a DCintra prediction mode (DC mode), and the directional intra predictionmodes may include second to 66^(th) intra prediction modes.

Meanwhile, the intra prediction mode may further include across-component linear model (CCLM) mode for chroma samples in additionto the above-described intra prediction modes. The CCLM mode may besplit into L_CCLM, T_CCLM, LT_CCLM according to whether left samples,upper samples or both thereof is considered for LM parameter derivationand may apply only to a chroma component.

For example, the intra prediction mode may be, for example, indexed asshown in the following table.

TABLE 1 Intra prediction mode Associated name 0 INTRA_PLANAR 1 INTRA_DC 2 . . . 66 INTRA_ANGULAR2 . . . INTRA_ANGULAR66 81 . . . 83INTRA_LT_CCLM, INTRA_L_CCLM, INTRA_T_CCLM

FIG. 8 b shows an intra prediction direction according to anotherembodiment of the present disclosure. In FIG. 8 b, a dotted-linedirection shows a wide angle mode applying only to a non-square block.As shown in FIG. 8 b, in order to capture any edge direction presentedin natural video, the intra prediction mode according to an embodimentmay include two non-directional intra prediction modes and 93directional intra prediction modes. The non-directional intra predictionmodes may include a planar mode and a DC mode, and the directional intraprediction modes may include second to 80^(th) and −1^(st) to −14^(th)intra prediction modes, as denoted by arrow of FIG. 8 b. The planar modemay be denoted by INTRA_PLANAR, and the DC mode may be denoted byINTRA_DC. In addition, the directional intra prediction mode may bedenoted by INTRA_ANGULAR-14 to INTRA_ANGULAR-1 and INTRA_ANGULAR2 toINTRA_ANGULAR80. In addition, the intra prediction technique informationmay be implemented in various forms. For example, the intra predictiontechnique information may include intra prediction type indexinformation indicating one of the intra prediction techniques. Asanother example, the intra prediction technique information may includeat least one of reference sample line information (e.g.,intra_luma_ref_idx) indicating whether to apply MRL to the current blockand, if applied, which reference sample line is used, ISP flaginformation (e.g., intra_subpartitions_mode_flag) indicating whether toapply ISP to the current block, ISP type information (e.g.,intra_subpartitions_split_flag) indicating the split type ofsub-partitions when applying ISP, flag information indicating whether toapply PDPC, or flag information indicating whether to apply LIP. In thepresent disclosure, ISP flag information may be referred to as an ISPapplication indicator.

The intra prediction mode information and/or the intra predictiontechnique information may be encoded/decoded through the coding methoddescribed in the present disclosure. For example, the intra predictionmode information and/or the intra prediction technique information maybe encoded/decoded through entropy coding (e.g., CABAC, CAVLC) based ona truncated (rice) binary code.

Before determining whether to apply a predetermined intra predictiontechnique to a current block, whether the corresponding intra predictiontechnique is available for the current block may be first determined.For example, whether the corresponding intra prediction technique isavailable for the current block may be determined based on the codingparameters of the current block. In this case, the coding parameters mayinclude a size (width and/or height) of the current block, a position ofthe current block, a color component of the current block or whether toapply another intra prediction technique.

In addition, determination as to whether the corresponding intraprediction technique is available for the current bock may be performedbased on information signaled at a higher level of the current block,such as a sequence, a picture, a slice and a CTU. For example, wheninformation transmitted at a sequence level indicates that thepredetermined intra prediction technique is not available, it may bedetermined that the corresponding intra prediction technique is notavailable for blocks belonging to the corresponding sequence.

Upon determining that the predetermined intra prediction technique isavailable for the current block, the image encoding apparatus maydetermine whether to apply the corresponding intra prediction techniqueto the current block using various methods. For example, the imageencoding apparatus may determine whether to apply the correspondingintra prediction technique based on RDO.

The image encoding apparatus may encode and signal an intra predictiontechnique information in a bitstream. The intra prediction techniqueinformation may be encoded according to the corresponding intraprediction technique in various forms. For example, when the intraprediction technique is MRL, the intra prediction technique informationmay be an index (e.g., intra_luma_ref_idx) indicating a reference lineused for prediction of the current block among multiple reference lines.When the intra prediction technique is ISP, the intra predictiontechnique information may be flag information (e.g.,intra_subpartitions_mode_flag) indicating whether to apply ISP to thecurrent block. In addition, when applying ISP, the intra predictiontechnique information may further include information (e.g.,intra_subpartitions_split_flag) on a split direction. In addition, theintra prediction technique information may include flag informationindicating whether to apply PDPC, flag information indicating whether toapply LIP or flag information indicating whether to apply an LM mode.

In case it is determined that the predetermined intra predictiontechnique is available for the current block, the image decodingapparatus may determine whether to apply the corresponding intraprediction technique to the current block based on the signaled intraprediction technique information.

As another example, whether to apply the predetermined intra predictiontechnique to the current block may be implicitly derived by the imageencoding apparatus and the image decoding apparatus based on codingparameters for the current block in addition to the explicitly signaledintra prediction technique information. In this case, the codingparameter may include a size (width and/or height) of the current block,a position of the current block, a color component of the current blockor whether to apply another intra prediction technique.

Hereinafter, an ISP mode according to the present disclosure will bedescribed in detail.

The current block may include a luma component block (array) and achroma component block (array) corresponding thereto. In the presentdisclosure, the “current block” or the “luma component block” may mean a“luma component block of a current block”, and the “chroma componentblock” or the “corresponding chroma component block” may mean a “chromacomponent block of the current block”. In addition, the “luma componentblock” may be referred to as the term “luma block”, “luminance componentblock” or “luminance block”, and the “chroma component block” may bereferred to as the term “chroma block”, “chrominance component block” or“chrominance block”.

In conventional intra prediction, a current coding/decoding target block(current block) is regarded as one unit and thus coding/decoding isperformed without split. However, when applying an ISP mode, the currentblock is split in a horizontal or vertical direction to perform intraprediction encoding/decoding. In this case, encoding/decoding isperformed in units of split sub-partitions to generate a reconstructedsub-partition and the reconstructed sub-partition is used as a referenceblock of a next split sub-partition.

Whether ISP is available for the current block may be determined basedon the following conditions of table 2. The following conditions may bedetermined based on the luma component block of the current block. Thatis, in the following conditions, the width, height and position of thecurrent block may mean the width, height and position of the lumacomponent block of the current block, respectively.

TABLE 2 <ISP availability conditions> - intra_luma_ref_idx[ x0 ][ y0 ] == 0 - cbWidth <= MaxTbSizeY ∥ cbHeight <= MaxTbSizeY - cbWidth *cbHeight > MinTbSizeY * MinTbSizeY

For example, when all of the above conditions for the current block aresatisfied, it may be determined that ISP is available for the currentblock. After determining that ISP is available for the current block, itmay be determined whether to apply ISP to the current block. In the ISPavailability conditions, (x0, y0) is coordinates indicating the locationof the top-left sample of the current block. In addition,intra_luma_ref_idx[x0][y0] is information indicating a reference lineused for prediction of the current block. According to the ISPavailability conditions, when intra_luma_ref_idx is 0, that is, when areference line used for prediction of the current block is line 0 (areference line immediately adjacent to the current block), it may bedetermined that ISP is available for the current block. When a referenceline used for prediction of the current block is a line other than line0, it may be determined that ISP is not available for the current block.

In the ISP availability conditions, cbWidth and cbHeight correspond tothe width and height of the current block, respectively. In addition,MaxTbSizeY and MinTbSizeY may indicate a maximum transform size and aminimum transform size, respectively. As described above, the residualprocessing may include transform or inverse transform. In this case, asize of a transform block in which transform or inverse transform isavailable may be predefined or signaled through a bitstream. That is,the maximum transform size means a maximum size of a transform block inwhich transform or inverse transform may be performed. In addition, theminimum transform size means a minimum size of a transform block inwhich transform or inverse transform may be performed. For example, whenthe size of the current block is greater than the maximum transformsize, the current block may be split into two or more transform blocks.In addition, the current block may not be split into transform blockshaving a size less than the minimum transform size. The maximumtransform size and/or the minimum transform size may be predefined inthe image encoding apparatus and the image decoding apparatus or may bederived based on information signaled at a higher level of a block.

According to the ISP availability conditions, only when at least one ofcbWidth or cbHeight is equal to or less than MaxTbSizeY, it may bedetermined that ISP is available for the current block. That is, whenboth cbWidth and cbHeight are greater than MaxTbSizeY, it may bedetermined that ISP is not available for the current block. When cbWidthis greater than MaxTbSizeY, cbHeight is equal to or less than MaxTbSizeYand ISP applies to the current block, an ISP split direction may bedetermined as a vertical direction as described below. When cbHeight isgreater than MaxTbSizeY, cbWidth is equal to or less than MaxTbSizeY andISP applies to the current block, an ISP split direction may bedetermined as a horizontal direction as described below.

According to the ISP availability conditions, when cbWidth*cbHeight isgreater than MinTbSizeY*MinTbSizeY, it may be determined that ISP isavailable for the current block. cbWidth*cbHeight may mean the area ofthe current block or the number of samples included in the currentblock. For example, if MinTbSizeY is 4, it may be determined that ISP isavailable for the current block only when the number of samples includedin the current block is greater than 16(4*4).

FIG. 9 is a view illustrating a conventional method of performingencoding/decoding of a current block according to an ISP mode.

When a current block to be encoded/decoded is input (S910), it may bedetermined whether to apply ISP to the current block (S920).Determination of step S920 may include determining whether ISP isavailable for the current block and/or determining whether to apply ISPto the current block. The determination as to whether ISP is availablefor the current block may be made based on the above-described ISPavailability conditions. When ISP is available, the image encodingapparatus may determine whether to apply ISP to the current block basedon various methods as described above, and the result of determinationmay be encoded in a bitstream as intra prediction technique information.When ISP is available, the image decoding apparatus may determinewhether to apply ISP to the current block based on the signaled intraprediction technique information.

When ISP does not apply to the current block, TU tiling may be performedwith respect to the current block (S930). TU tiling means a process ofsplitting the current block into a plurality of transform blocks suchthat the width and height of the current block are equal to or less thana maximum transform size which is a transformable size. As describedabove, processing of the residual signal includes transform, and themaximum transform size means a maximum size of a transform block inwhich a transform process may be performed. Accordingly, when the widthor height of the current block is greater than the maximum transformsize, by splitting the current block through TU tiling, both the widthand height of the split block may be equal to or less than the maximumtransform size. For example, when the maximum transform size is a sizeof 64 samples and the current block has a size of 128×128, the currentblock may be split into four 64×64 blocks. Alternatively, when themaximum transform size is a size of 64 samples and the current block hasa size of 64×128 or 128×64, the current block may be split into two64×64 blocks. For example, the maximum transform size is a size of 64samples and the current block has a size of 64×64 or less, TU tiling maynot be performed.

Thereafter, encoding/decoding may be performed with respect to thecurrent block or each of the blocks split through the TU tiling step(S960). Encoding of step S960 may include intra prediction, residualprocessing and/or encoding of prediction information and residualinformation. Encoding of step S960 may include intra prediction,residual samples derivation and/or reconstructed block generation.

In step S920, when applying ISP to the current block, a split directionand the number of splits (the number of sub-partitions) may bedetermined (S940).

In step S920, the split direction may be derived based on information(e.g., intra_subpartitions_split_flag) signaled through a bitstream.Alternatively, the split direction may be implicitly derived based onthe size of the current block. For example, as described above, when thewidth of the current block is greater than the maximum transform size,the split direction of ISP may be derived in a vertical direction. Inaddition, when the height of the current block is greater than themaximum transform size, the split direction of ISP may be derived in ahorizontal direction. In addition, when both the width and height of thecurrent block are greater than the maximum transform size, restrictionmay be imposed such that ISP is not available for the current block. Forexample, when the maximum transform size is a size of 64 samples and thecurrent block to which ISP applies is a 128×64 block, the splitdirection of ISP is derived in a vertical direction and both the width(128/4) and height (64) of the sub-partition are determined as a maximumtransform size (64) or less. Similarly, when the maximum transform sizeis a size of 64 samples and the current block to which ISP applies is a64×128 block, the split direction of ISP is derived in a horizontaldirection and both the width (64) and height (128/4) of thesub-partition are determined as a maximum transform size (64) or less.As described above, when applying ISP to the current block, the currentblock is split such that both the width and height of sub-partitions isequal to or less than the maximum transform size. For this reason, TUtiling of step S930 does not need to be performed with respect to thecurrent block to which ISP applies.

In step S940, the number of splits may be implicitly derived based onthe size of the current block. Specifically, the current block may besplit according to the size of the current block, as shown in Table 3.

TABLE 3 Block size Number of splits 4 × 4 No split 4 × 8, 8 × 4 2 Allother cases 4

FIG. 10 a is a view illustrating a split example of ISP for a 4×8 blockor 8×4 block. As shown in FIG. 10 a, the 4×8 block or 8×4 block may besplit into two sub-partitions. When the current block is a 4×8 block andis split in a horizontal direction, the current block may be split intotwo 4×4 sub-partitions. When the current block is a 4×8 block and issplit in a vertical direction, the current block may be split into two2×8 sub-partitions. When the current block is an 8×4 block and is splitin a horizontal direction, the current block may be split into two 8×2sub-partitions. When the current block is an 8×4 block and is split in avertical direction, the current block may be split into two 4×4sub-partitions.

FIG. 10 b is a view illustrating a split example of ISP for a blockhaving a size of 8×8 or more.

As shown in FIG. 10 b, a block having a size of 8×8 or more may be splitinto four sub-partitions. When the current block is a W×H block, both Wand H are 8 or more and the current bock is split in a horizontaldirection, the current block may be split into four W×(H/4)sub-partitions. When the current block is a W×H block, both W and H are8 or more, and the current block is split in a vertical direction, thecurrent block may be split into four (W/4)×H sub-partitions.

Referring to FIG. 9 again, when the split direction and the number ofsplits are determined in step S940, the luma component block of thecurrent block may be split based on this (S950).

Thereafter, encoding/decoding may be performed with respect to eachsplit sub-partition (S960). As described above, encoding of step S960may include intra prediction, residual processing and/or encoding ofprediction information and residual information. In addition, decodingof step S960 may include intra prediction, residual samples derivationand/or reconstructed block generation. Specifically, when applying ISP,the intra prediction mode for the current block equally applies to thesub-partitions, and a neighboring reference sample is derived and usedin units of sub-partitions, thereby increasing intra predictionperformance That is, when applying ISP, a residual sample processingprocedure is performed in units of sub-partitions. In other words, intraprediction samples are derived for each sub-partition, and a residualsignal (residual samples) for the corresponding sub-partition is addedthereto, thereby obtaining reconstructed samples. The residual signal(residual samples) may be derived through a dequantization/inversetransform procedure based on residual information (quantized transformcoefficient information or residual coding syntax) in theabove-described bitstream. That is, prediction samples derivation andresidual samples derivation for a first sub-partition may be performedand reconstructed samples for the first sub-partition may be derivedbased on this. In this case, when deriving prediction samples for asecond sub-partition, some of the reconstructed samples in the firstsub-partition (e.g., reconstructed samples adjacent to the left or upperside of the second sub-partition) may be used as neighboring referencesamples for the second sub-partition. Similarly, prediction samplesderivation and residual samples derivation for the second sub-partitionmay be derived, and reconstructed samples for the second sub-partitionmay be derived based on this. In this case, when prediction samples fora third sub-partition are derived, some of the reconstructed samples inthe second sub-partition (e.g., reconstructed samples adjacent to theleft or upper side of the third sub-partition) may be used asneighboring reference samples for the third sub-partition. Similarly,some of the reconstructed samples in the third sub-partition may be usedas neighboring reference samples for a fourth sub-partition.

The order of encoding/decoding among a plurality of sub-partitions isfrom top to bottom when the split direction is a horizontal directionand is from left to right when the split direction is a verticaldirection. For example, in FIG. 10 b, when the split direction is ahorizontal direction, sub-partitions may be sequentially encoded/decodedfrom the uppermost sub-partition to the lowermost sub-partition. Inaddition, when the split direction is a vertical direction,sub-partitions may be sequentially encoded/decoded from the leftmostsub-partition to the rightmost sub-partition.

When applying ISP to the current block, in order to reduce codingcomplexity, an MPM list is generated according to each split method(horizontal split and vertical split) and a suitable prediction modeamong prediction modes in the generated MPM list is compared in terms ofrate distortion optimization (RDO) to generate an optimal mode. Inaddition, when multi-reference line (MRL) intra prediction is used,restriction may be imposed such that the above-described ISP is notused. That is, when a 0^(th) reference line is used (e.g.,intra_luma_ref_idx==0), ISP is applicable. In addition, when applyingISP, restriction may be imposed such that the above-described PDPC isnot used. That is, when applying ISP, PDPC may not be used.

When applying ISP, as intra prediction technique information,information (intra_subpartitions_mode_flag) indicating whether to applyISP may be transmitted in units of current blocks, and if the currentblock uses ISP (e.g., when intra_subpartitions_mode_flag is 1),information (intra_subpartitions_split_flag) on the split method(horizontal split or vertical split) may be transmitted.

According to the conventional method described with reference to FIG. 9, in a single tree structure in which the luma component block of thecurrent block and the chroma component block of the current block aresplit into the same tree structure, when applying ISP to the currentblock, the luma component block is split into a plurality ofsub-partitions but ISP does not apply to the chroma component block. Inthis case, the width or height of the chroma component block becomesgreater than the maximum transform size according to the chroma formatand thus transform or inverse transform of the chroma component blockmay become impossible.

Hereinafter, a relationship between the size of the luma component blockand the size of the chroma component block according to the chromaformat will be described.

FIG. 11 is a view illustrating a relationship between a luma componentblock (luma component array) and a chroma component block (chromacomponent array) according to a chroma format.

A source or coded picture/image may include a luma component (Y) blockand two chroma component (cb, cr) blocks. That is, one pixel of thepicture/image may include a luma sample and two chroma samples (cb, cr).A color format may represent a configuration format of a luma sample andchroma samples (cb, cr), and may be referred to as a chroma format. Thechroma format may be predefined or may be adaptively signaled. Forexample, the chroma format may be signaled based on at least one ofchroma_format_idc or separate_colour_plane_flag as shown in Table 4.

TABLE 4 chroma_format_idc separate_colour_plane_flag ChromaArrayTypeChroma format SubWidthC SubHeightC 0 0 0 Monochrome 1 1 1 0 1 4:2:0 2 22 0 2 4:2:2 2 1 3 0 3 4:4:4 1 1 3 1 0 4:4:4 1 1

In Table 4 above, chroma_format_idc is information indicating the formatof a luma sample and a chroma sample corresponding thereto, andseparate_colour_plane_flag is information indicating three colorcomponents Y, cb and cr are separately encoded in a 4:4:4 chroma format.In Table 4 above, when chroma_format_idc is 0, the chroma formatcorresponds to monochrome, and the current block does not include thechroma component block and includes only the luma component block.

In Table 4 above, when chroma_format_idc is 1, the chroma formatcorresponds to a 4:2:0 chroma format, and the width and height of thechroma component block respectively correspond to half the width andhalf the height of the luma component block. FIG. 1 la shows thelocation relationship between the luma sample and the chroma sample in a4:2:0 chroma format.

In Table 4 above, when chroma_format_idc is 2, the chroma formatcorresponds to a 4:2:2 chroma format, the width of the chroma componentblock respectively correspond to half the width of the luma componentblock, and the height of the chroma component block is equal to that ofthe luma component block. FIG. 11 b shows the location relationshipbetween the luma sample and the chroma sample in a 4:2:2 chroma format.

In Table 4 above, when chroma_format_idc is 3, the chroma formatcorresponds to a 4:4:4 chroma format, and the width and height of thechroma component block respectively correspond to the width and heightof the luma component block. FIG. 11 c shows the location relationshipbetween the luma sample and the chroma sample in a 4:4:4 chroma format.

In Table 4 above, SubWidthC and SubHeightC represent a ratio of the lumasample and the chroma sample. For example, when the width and height ofthe luma component block are respectively CbWidth and CbHeight, thewidth and height of the chroma component block corresponding thereto maybe derived as (CbWidth/SubwidthC) and (CbHeight/SubHeightC),respectively.

As described with reference to FIG. 11 , the size of the chromacomponent block corresponding to the luma component block of the currentblock may vary according to the chroma format.

FIG. 12 is a view illustrating the size of a chroma component blockaccording to a chroma format when a luma component block is a 64×128block. As shown in FIG. 12 , a chroma component block corresponding to a64×128 luma component block may be a 32×64 block in a 4:2:0 chromaformat, a 32×128 block in a 4:2:2 chroma format, and a 64×128 block in a4:4:4 chroma format.

As described above, according to the conventional method described withreference to FIG. 9 , in a single tree structure in which the lumacomponent block and the chroma component block are split into the sametree structure, when applying ISP to the current block, the lumacomponent block is split into a plurality of sub-partitions, but thechroma component block is not split. For example, when the lumacomponent block of the current block to which ISP applies is a 64×128block, since a split direction is determined as a horizontal direction,the luma component block is split into four 64×32 sub-partitions, andboth the width (64) and height (32) of each sub-partition is equal to orless than a maximum transform size (64). Accordingly, transform orinverse transform of each of the sub-partitions for the luma componentblock may be performed. However, as shown in FIG. 12 , the chromacomponent block is a 32×64 block in a 4:2:0 chroma format, a 32×128block in a 4:2:2 chroma format or a 64×128 block in a 4:4:4 chromaformat, and, for example, in the 4:2:2 chroma format and the 4:4:4chroma format, since the height (128) of the chroma component block isgreater than the maximum transform size (64), transform or inversetransform of the chroma component block is impossible.

Hereinafter, various embodiments of the present disclosure for solvingthe above problems will be described in detail.

Embodiment #1

In Embodiment #1 of the present disclosure, in order to solve theconventional problems, when applying ISP to the current block, ISPadaptively applies to the chroma component block based on the chromaformat and/or the size of the chroma component block. According toEmbodiment #1 of the present disclosure, for example, when applying ISPto the chroma component block, the split direction and the number ofsplits determined with respect to the luma component block is equallyapplicable to the chroma component block.

FIG. 13 is a flowchart illustrating an ISP application method accordingto an embodiment of the present disclosure.

FIG. 14 is a view illustrating an example of splitting a chromacomponent block according to the ISP application method of FIG. 13 .

When a current block to be encoded/decoded is input (S1310), it may bedetermined whether to apply ISP to the current block (S1320).Determination of step S1320 may include determining whether ISP isavailable for the current block and/or whether to apply ISP to thecurrent block. The determination as to whether ISP is available for thecurrent block may be performed based on the above-described ISPavailability conditions. When ISP is available, the image encodingapparatus may determine whether to apply ISP to the current block basedon various methods as described above, and the result of determinationmay be encoded in a bitstream as intra prediction technique information.When ISP is available, the image decoding apparatus may determinewhether to apply ISP to the current block based on signaled intraprediction technique information.

When ISP does not apply to the current block, TU tiling may be performedwith respect to the current block (S1330). Thereafter, encoding/decodingmay be performed with respect to the current block or each of the blockssplit through the TU tiling step (S1390). Steps S1330 and S1390 areequal to steps S930 and S960 of FIG. 9 and thus a detailed descriptionthereof will be omitted.

In step S1320, when applying ISP to the current block, a split directionand the number of splits (the number of sub-partitions) may bedetermined (S1340). The description of the determination of the splitdirection and the number of splits is equal to the description of FIG. 9and thus will be omitted.

When the split direction and the number of splits are determined in stepS1340, the luma component block of the current block may be split basedon this (S1350).

Thereafter, it may be determined whether to apply ISP to the chromacomponent block corresponding to the luma component block (S1360). Thedetermination of step S1360 may be performed by the chroma format and/orby comparison between the width and height of the chroma component blockand the maximum transform size.

In the case of the 4:2:0 chroma format, as described below, the widthand height of the chroma component block is equal to or less than themaximum transform size. Accordingly, in the case of the 4:2:0 chromaformat, without comparing the width and height of the chroma componentblock with the maximum transform size, it may be determined that ISPdoes not apply to the chroma component block.

In the case of the 4:2:2 chroma format or the 4:4:4 chroma format, asdescribed below, the width or height of the chroma component block maybe greater than the maximum transform size. Accordingly, in the case ofthe 4:2:2 chroma format or the 4:4:4 chroma format, the width and heightof the chroma component block may be compared with the maximum transformsize, and, when the width or height of the chroma component block isgreater than the maximum transform size, it may be determined that ISPapplies to the chroma component block.

As shown in FIG. 14 , when the luma component block is a 64×128 block,the chroma component block corresponding thereto is a 32×64 block in the4:2:0 chroma format, a 32×128 block in the 4:2:2 chroma format or a64×128 block in the 4:4:4 chroma format.

In the example shown in FIG. 14 , it can be seen that both the width(32) and height (64) of the chroma component block of the 4:2:0 chromaformat are equal to or less than the maximum transform size (64).Accordingly, in the case of the 4:2:0 chroma format, it may bedetermined that ISP does not apply to a 32×64 chroma component blockwithout additional size comparison.

In contrast, in the case of the 4:2:2 chroma format or the 4:4:4 chromaformat, additionally, the width and height of the chroma component blockmay be compared with the maximum transform size. In the example shown inFIG. 14 , since the width (128) of the 32×128 chroma component block ofthe 4:2:2 chroma format and the width (128) of the 64×128 chromacomponent block of the 4:4:4 chroma format is greater than the maximumtransform size (64), it may be determined that ISP applies to the chromacomponent block.

As a modification of Embodiment #1, regardless of the chroma format,based on whether both the width and height of the chroma component blockare equal to or less than the maximum transform size, it may bedetermined whether to apply ISP to the chroma component block. Forexample, when both the width and height of the chroma component blockare equal to or less than the maximum transform size, it may bedetermined that ISP does not apply to the chroma component block. Inaddition, when the width or height of the chroma component block isgreater than the maximum transform size, it may be determined that ISPapplies to the chroma component block.

Upon determining that ISP applies to the chroma component block, thechroma component block may be split (S1370). Split of the chromacomponent block may be performed based on the split direction and thenumber of splits determined in step S1340. Specifically, the splitdirection and the number of splits for the chroma component block may bedetermined equally to the split direction and the number of splits forthe luma component block. According to Embodiment #1 of the presentdisclosure, in the example shown in FIG. 14 , the luma component blockis split into four sub-partitions in a horizontal direction.Accordingly, the 32×128 chroma component block of the 4:2:2 chromaformat and the 64×128 chroma component block of the 4:4:4 chroma formatmay be split into four sub-partitions in the horizontal directionequally to the luma component block.

Thereafter, each of the split sub-partitions may be encoded/decoded(S1390). In this case, encoding/decoding of step S1390 may be performedwith respect to each sub-partition of the luma component block and eachsub-partition of the chroma component block. Step S1390 is equal to stepS960 and thus a detailed description thereof will be omitted.

In step S1360, upon determining that ISP does not apply to the chromacomponent, the chroma component block is not split (S1380), and eachsub-partition of the luma component block and the unsplit chromacomponent block may be encoded/decoded (S1390). Step S1390 is equal tostep S960 and thus a detailed description thereof will be omitted.According to Embodiment #1 of the present disclosure, in the exampleshown in FIG. 14 , the 32×64 chroma component block of the 4:2:0 chromaformat may not be split.

According to Embodiment #1 of the present disclosure, when the width orheight of the chroma component block of the current block to which ISPapplies is greater than the maximum transform size, it may be possibleto solve that problem that transform or inverse transform of the chromacomponent block is impossible, by applying ISP to the chroma componentblock. In addition, according to Embodiment #1 of the presentdisclosure, when the chroma format of the current block to which ISPapplies is 4:2:0 or both the width and height of the chroma componentblock of the current block to which ISP applies are equal to or lessthan the maximum transform size, ISP does not apply to the chromacomponent block. Therefore, since it is possible to avoid the case whereISP unnecessarily applies to the chroma component block, the amount ofcomputation of encoding/decoding may be reduced. In addition, accordingto Embodiment #1 of the present disclosure, since the split directionand the number of splits of ISP for the chroma component block aredetermined equally to the split direction of the number of splits of ISPfor the luma component block, the split direction and the number ofsplits of ISP for the chroma component block does not need to beseparately signaled or derived.

Embodiment #2

In Embodiment #2 of the present disclosure, in order to solve theconventional problem, when ISP applies to the current block, ISP alsoapplies to the chroma component block. According to Embodiment #2 of thepresent disclosure, the split direction and the number of splits for thechroma component block may be determined based on the split directionand the number of splits determined for the luma component block.

FIG. 15 is a flowchart illustrating an ISP application method accordingto another embodiment of the present disclosure.

FIG. 16 is a view illustrating an example of splitting a chromacomponent block according to the ISP application method of FIG. 15 .

When a current block to be encoded/decoded is input (S1510), it may bedetermined whether to apply ISP to the current block (S1520).Determination of step S1520 may include determining whether ISP isavailable for the current block and/or whether to apply ISP to thecurrent block. The determination as to whether ISP is available for thecurrent block may be performed based on the above-described ISPavailability conditions. When ISP is available, the image encodingapparatus may determine whether to apply ISP to the current block basedon various methods as described above, and the result of determinationmay be encoded in a bitstream as intra prediction technique information.When ISP is available, the image decoding apparatus may determinewhether to apply ISP to the current block based on signaled intraprediction technique information.

When ISP does not apply to the current block, TU tiling may be performedwith respect to the current block (S1530). Thereafter, encoding/decodingmay be performed with respect to the current block or each of the blockssplit through the TU tiling step (S1570). Steps S1530 and S1570 areequal to steps S930 and S960 of FIG. 9 and a detailed descriptionthereof will be omitted.

In step S1520, when applying ISP to the current block, a split directionand the number of splits (the number of sub-partitions) may bedetermined (S1540). The description of the determination of the splitdirection and the number of splits is equal to the description of FIG. 9and thus will be omitted.

The luma component block of the current block may be split based on thesplit direction and the number of splits determined in step S1540(S1550).

Thereafter, the chroma component block of the current block may be splitbased on the split direction and the number of splits determined in stepS1540 (S1560). Specifically, the split direction for the chromacomponent block may be determined equally to the split direction for theluma component block. In addition, the number of splits for the chromacomponent block may be determined based on the number of splits for theluma component block. For example, when the number of splits for theluma component block is N, the number of splits for the chroma componentblock may be determined as N/n. In Embodiment #2 of the presentdisclosure, for example, n may be 2, but is not limited thereto and nmay be any integer. According to Embodiment #2, in the example shown atthe top of FIG. 16 , a 16×16 luma component block may be split into foursub-partitions in a horizontal direction. In this case, the splitdirection of the corresponding chroma component block is a horizontaldirection, and the number of splits may be determined as 2. That is, thecorresponding chroma component block may be split into twosub-partitions in the horizontal direction regardless of the chromaformat and/or the size of the chroma component block. In addition, inthe example shown at the bottom of FIG. 16 , a 4×8 luma component blockis split into two sub-partitions in a vertical direction. In this case,the split direction of the corresponding chroma component block is avertical direction and the number of splits may be determined as 1. Inthis case, since the chroma component block is not substantially split,it can be seen that ISP does not apply to the chroma component block.

Thereafter, each sub-partition or unsplit chroma component block may beencoded/decoded (S1570). In this case, encoding/decoding of step S1570may be performed with respect to each sub-partition of the lumacomponent block and each sub-partition of the chroma component block orthe chroma component block. Step S1570 is equal to step S960 and thus adetailed description thereof will be omitted.

In Embodiment #2 of the present disclosure, when the size of the maximumcoding unit is 128×128, the maximum transform size is 64 which is halfthe width and height of the maximum coding unit and both the width andheight of the current block are greater than the maximum transform size,ISP does not apply to the current block. According to Embodiment #2 ofthe present disclosure, since the width and height of the chromacomponent block of the current block to which ISP applies or thesub-partition of the chroma component block are always equal to or lessthan the maximum transform size, the problem that transform or inversetransform of the chroma component block is impossible may be solved. Inaddition, according to Embodiment #2 of the present disclosure, sincethe determination as to whether to apply ISP to the chroma componentblock of the current block (for example, determination of S1360) doesnot need to be performed, the amount of computation of encoding/decodingmay be reduced. In addition, according to Embodiment #2 of the presentdisclosure, since the number of sub-partitions for the chroma componentblock is determined as 2 or 1, it is possible to simplify theencoding/decoding process of the chroma component block. In addition,according to Embodiment #2 of the present disclosure, since the splitdirection and the number of splits of ISP for the chroma component blockare determined equally to the split direction and (the number ofsplits/2) of ISP for the luma component block, the split direction andthe number of splits of ISP for the chroma component block does not needto be separately signaled or derived.

Embodiment #3

In Embodiment #3 of the present disclosure, in order to solve theconventional problem, the width and height of the chroma component blockare equal to or less than the maximum transform size by changing the ISPavailability conditions.

According to the conventional ISP method described with reference toFIG. 9 , when both the width and height of the current block are greaterthan the maximum transform size, it is determined that ISP does notapply to the current block. That is, when one of the width or height ofthe current block is greater than the maximum transform size and theother is equal to or less than the maximum transform size, it isdetermined that ISP applies to the current block.

In Embodiment #3 of the present disclosure, the above-described ISPavailability conditions are modified as following table 5.

TABLE 5 <Modified ISP availability conditions> - intra_luma_ref_idx[ x0][ y0 ] = = 0 - cbWidth <= MaxTbSizeY && cbHeight <= MaxTbSizeY -cbWidth * cbHeight > MinTbSizeY * MinTbSizeY

Among the modified ISP availability conditions, a condition in whichintra_luma_ref_idx[x0][y0] is 0 and a condition in whichcbWidth*cbHeight is greater than MinTbSizeY*MinTbSizeY are equal to theexisting ISP availability conditions. According to the modified ISPavailability conditions, only when both cbWidth and cbHeight of thecurrent block are equal to or less than MaxTbSizeY, it may be determinedthat ISP is available for the current block.

According to Embodiment #3 of the present disclosure to which themodified ISP availability conditions apply, only when both the width andheight of the current block are equal to or less than the maximumtransform size, it is determined that ISP applies to the current block,and intra_subpartitions_mode_flag indicating whether to apply ISP may betransmitted.

According to Embodiment #3 of the present disclosure, since both thewidth and height of the current block are equal to or less than themaximum transform size, the width and/or height of the chroma sampleblock corresponding to the luma sample block of the current block arealways equal to or less than the maximum transform size regardless ofthe chroma format. Accordingly, it is possible to solve the problem ofthe conventional ISP method described with reference to FIG. 9 , byapplying the modified ISP availability conditions.

A method according to Embodiment #3 of the present disclosure may beimplemented by modifying the ISP availability conditions of theconventional method. Accordingly, the method according to Embodiment #3of the present disclosure may be performed equally to the flowchartshown in FIG. 9 except that only the condition for determining whetherISP is available in step S920 is different.

FIG. 17 is a view illustrating an example of the structure of abitstream in which a modified ISP availability condition is reflected.

The rectangular box of FIG. 17 is related to signaling of information onISP of intra prediction technique information of a current coding unit.As shown in FIG. 17 , when the above-described modified ISP availabilityconditions are satisfied, intra_subpartitions_mode_flag regardingwhether to apply ISP may be transmitted. intra_subpartitions_mode_flagis information indicating whether to apply ISP to the current codingunit. When the ISP availability conditions are not satisfied,intra_subpartitions_mode_flag is not transmitted, and it may bedetermined that ISP does not apply to the current coding unit.

The modified ISP availability conditions may include one or moreconditions, and one or more conditions included in the modified ISPavailability conditions are not limited to the above examples. That is,some conditions may be omitted or other conditions may be additionallyincluded within the scope of the technical idea according to the presentdisclosure.

In addition, as shown in FIG. 17 , when ISP applies to the currentblock, that is, when intra_subpartitions_mode_flag is 1, information(intra_subpartitions_split_flag) indicating the split direction may betransmitted.

In the example shown in FIG. 17 , only when the condition of cbWidth<=MaxTbSizeY && cbHeight<=MaxTbSizeY is satisfied,intra_subpartitions_mode_flag may be transmitted. That is, when thetransmitted intra_subpartitions_mode_flag is 1, it can be seen that thecondition of cbWidth<=MaxTbSizeY && cbHeight<=MaxTbSizeY I is alreadysatisfied. Accordingly, the condition of FIG. 17 related to whether totransmit intra_sinpartitions_split_flag may be changed as shown in FIG.18 .

FIG. 18 is a view illustrating another example of the structure of abitstream in which a modified ISP availability condition is reflected.

The rectangular box of FIG. 18 is related to signaling of information onISP of intra prediction technique information of the current codingunit. As shown in FIG. 18 , intra_subpartitions_mode_flag may besignaled when the above-described modified ISP availability conditionsare satisfied, and intra_sinpartitions_split_flag may be signaled when acondition in which intra_subpartitions_mode_flag is 1 is satisfied.

According to Embodiment #3 of the present disclosure, both the width andheight of the current block to which ISP applies are equal to or lessthan the maximum transform size. Accordingly, since both the width andheight of the chroma component block of the current block are equal toor less than the maximum transform size regardless of the chroma format,the problem that transform or inverse transform of the chroma componentblock is impossible may be solved. In addition, according to Embodiment#3 of the present disclosure, since the determination as to whether toapply ISP to the chroma component block of the current block (forexample, determination of S1360) and ISP for the chroma component blockdo not need to be performed, the amount of computation ofencoding/decoding may be reduced. In addition, according to Embodiment#3 of the present disclosure, since the conventional encoding/decodingprocess is not significantly changed, the conventional problem can besolved without significantly encoding/decoding complexity.

While the exemplary methods of the present disclosure described aboveare represented as a series of operations for clarity of description, itis not intended to limit the order in which the steps are performed, andthe steps may be performed simultaneously or in different order asnecessary. In order to implement the method according to the presentdisclosure, the described steps may further include other steps, mayinclude remaining steps except for some of the steps, or may includeother additional steps except for some steps.

In the present disclosure, the image encoding apparatus or the imagedecoding apparatus that performs a predetermined operation (step) mayperform an operation (step) of confirming an execution condition orsituation of the corresponding operation (step). For example, if it isdescribed that predetermined operation is performed when a predeterminedcondition is satisfied, the image encoding apparatus or the imagedecoding apparatus may perform the predetermined operation afterdetermining whether the predetermined condition is satisfied.

The various embodiments of the present disclosure are not a list of allpossible combinations and are intended to describe representativeaspects of the present disclosure, and the matters described in thevarious embodiments may be applied independently or in combination oftwo or more.

Various embodiments of the present disclosure may be implemented inhardware, firmware, software, or a combination thereof. In the case ofimplementing the present disclosure by hardware, the present disclosurecan be implemented with application specific integrated circuits(ASICs), Digital signal processors (DSPs), digital signal processingdevices (DSPDs), programmable logic devices (PLDs), field programmablegate arrays (FPGAs), general processors, controllers, microcontrollers,microprocessors, etc.

In addition, the image decoding apparatus and the image encodingapparatus, to which the embodiments of the present disclosure areapplied, may be included in a multimedia broadcasting transmission andreception device, a mobile communication terminal, a home cinema videodevice, a digital cinema video device, a surveillance camera, a videochat device, a real time communication device such as videocommunication, a mobile streaming device, a storage medium, a camcorder,a video on demand (VoD) service providing device, an OTT video (over thetop video) device, an Internet streaming service providing device, athree-dimensional (3D) video device, a video telephony video device, amedical video device, and the like, and may be used to process videosignals or data signals. For example, the OTT video devices may includea game console, a blu-ray player, an Internet access TV, a home theatersystem, a smartphone, a tablet PC, a digital video recorder (DVR), orthe like.

FIG. 19 is a view showing a content streaming system, to which anembodiment of the present disclosure is applicable.

As shown in FIG. 19 , the content streaming system, to which theembodiment of the present disclosure is applied, may largely include anencoding server, a streaming server, a web server, a media storage, auser device, and a multimedia input device.

The encoding server compresses content input from multimedia inputdevices such as a smartphone, a camera, a camcorder, etc. into digitaldata to generate a bitstream and transmits the bitstream to thestreaming server. As another example, when the multimedia input devicessuch as smartphones, cameras, camcorders, etc. directly generate abitstream, the encoding server may be omitted.

The bitstream may be generated by an image encoding method or an imageencoding apparatus, to which the embodiment of the present disclosure isapplied, and the streaming server may temporarily store the bitstream inthe process of transmitting or receiving the bitstream.

The streaming server transmits the multimedia data to the user devicebased on a user's request through the web server, and the web serverserves as a medium for informing the user of a service. When the userrequests a desired service from the web server, the web server maydeliver it to a streaming server, and the streaming server may transmitmultimedia data to the user. In this case, the content streaming systemmay include a separate control server. In this case, the control serverserves to control a command/response between devices in the contentstreaming system.

The streaming server may receive content from a media storage and/or anencoding server. For example, when the content are received from theencoding server, the content may be received in real time. In this case,in order to provide a smooth streaming service, the streaming server maystore the bitstream for a predetermined time.

Examples of the user device may include a mobile phone, a smartphone, alaptop computer, a digital broadcasting terminal, a personal digitalassistant (PDA), a portable multimedia player (PMP), navigation, a slatePC, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smartglasses, head mounted displays), digital TVs, desktops computer, digitalsignage, and the like.

Each server in the content streaming system may be operated as adistributed server, in which case data received from each server may bedistributed.

The scope of the disclosure includes software or machine-executablecommands (e.g., an operating system, an application, firmware, aprogram, etc.) for enabling operations according to the methods ofvarious embodiments to be executed on an apparatus or a computer, anon-transitory computer-readable medium having such software or commandsstored thereon and executable on the apparatus or the computer.

INDUSTRIAL APPLICABILITY

The embodiments of the present disclosure may be used to encode ordecode an image.

1-15. (canceled)
 16. An image decoding method performed by an imagedecoding apparatus, the image decoding method comprising: determining aprediction mode of a current block based on prediction mode informationof the current block; determining whether intra sub-partitions (ISP) isavailable for the current block based on the prediction mode of thecurrent block being an intra prediction mode; decoding ISP modeinformation for the current block, based on the ISP being available forthe current block; and generating a prediction sample for the currentblock based on the ISP mode information, wherein the ISP modeinformation includes information on an ISP split direction, wherein itis determined that the ISP is available for the current block based onboth a width and a height of a luma component block of the current blockbeing equal to or less than a predetermined threshold.
 17. The imagedecoding method of claim 16, wherein the predetermined threshold is amaximum transform size.
 18. The image decoding method of claim 16,wherein the generating the prediction sample for the current block basedon the ISP mode information comprises: generating a prediction samplefor a luma component block by applying the ISP to the luma componentblock of the current block; determining whether to apply the ISP to achroma component block of the current block; and generating a predictionsample for the chroma component block by applying the ISP to the chromacomponent block, upon determining that the ISP applies to the chromacomponent block of the current block.
 19. The image decoding method ofclaim 18, wherein the determining whether to apply the ISP to the chromacomponent block of the current block is performed based on at least oneof a size of the chroma component block or a chroma format of thecurrent block.
 20. The image decoding method of claim 19, wherein, whenthe chroma format of the current block is 4:2:2 or 4:4:4 and at leastone of a width or height of the chroma component block is greater than amaximum transform size, it is determined that the ISP applies to thechroma component block of the current block.
 21. The image decodingmethod of claim 18, wherein, when the ISP applies to the chromacomponent block, a split direction and a number of sub-partitions forthe chroma component block are determined based on a split direction anda number of sub-partitions for the luma component block.
 22. The imagedecoding method of claim 21, wherein the split direction and the numberof sub-partitions for the chroma component block are equal to the splitdirection and the number of sub-partitions for the luma component block,respectively.
 23. The image decoding method of claim 16, wherein thegenerating the prediction sample for the current block based on the ISPmode information comprises: generating a prediction sample for a lumacomponent block by applying the ISP to the luma component block of thecurrent block; and generating a prediction sample for a chroma componentblock by applying the ISP to the chroma component block of the currentblock.
 24. The image decoding method of claim 23, wherein a splitdirection and a number of sub-partitions for the chroma component blockare determined based on a split direction and a number of sub-partitionsfor the luma component block.
 25. The image decoding method of claim 24,wherein the split direction and the number of sub-partitions for thechroma component block are equal to the split direction and (the numberof sub-partitions/2) of the luma component block, respectively.
 26. Animage encoding method performed by an image encoding apparatus, theimage encoding method comprising: determining a prediction mode of acurrent block; determining whether intra sub-partitions (ISP) isavailable for the current block based on the prediction mode of thecurrent block being an intra prediction mode; determining whether toapply the ISP to the current block, based on the ISP being available forthe current block; generating a prediction sample for the current blockby applying the ISP to the current block, based on the ISP beingdetermined to be applied to the current block; and encoding predictionmode information of the current block and ISP mode information, whereinthe ISP mode information includes information on an ISP split direction,and wherein it is determined that the ISP is available for the currentblock based on both a width and a height of a luma component block ofthe current block being equal to or less than a predetermined threshold.27. A non-transitory computer-readable medium storing a bitstreamgenerated by an image encoding method, the image encoding methodcomprising: determining a prediction mode of a current block;determining whether intra sub-partitions (ISP) is available for thecurrent block based on the prediction mode of the current block being anintra prediction mode; determining whether to apply the ISP to thecurrent block, based on the ISP being available for the current block;generating a prediction sample for the current block by applying the ISPto the current block, based on the ISP being determined to be applied tothe current block; and encoding prediction mode information of thecurrent block and ISP mode information, wherein the ISP mode informationincludes information on an ISP split direction, and wherein it isdetermined that the ISP is available for the current block based on botha width and a height of a luma component block of the current blockbeing equal to or less than a predetermined threshold.